Scalable Graph Convolutional Network Training on Distributed-Memory Systems

Graph Convolutional Networks (GCNs) are extensively utilized for deep learning on graphs. The large data sizes of graphs and their vertex features make scalable training algorithms distributed memory systems necessary. Since the convolution operation induces irregular access patterns, designing a memory- communication-efficient parallel algorithm GCN poses unique challenges. We propose highly that scales to processor counts. In our solution, adjacency vertex-feature matrices partitioned among processors. exploit vertex-partitioning graph use non-blocking point-to-point communication operations between processors better scalability. To further minimize parallelization overheads, we introduce sparse matrix partitioning scheme based hypergraph model full-batch training. also novel stochastic encode expected volume in mini-batch show merits model, previously unexplored training, over standard which does not accurately costs. Experiments performed real-world datasets demonstrate proposed achieve considerable speedups alternative solutions. optimizations achieved costs become even more pronounced at high scalability with many performance benefits preserved deeper GCNs having layers as well billion-scale